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Executive Summary 


The 21st Century Community Learning Centers (21st CCLC) program, funded through 
the U.S. Department of Education, provides academic enrichment opportunities outside of the 
regular school day to help students meet state and local performance standards in core academic 
subjects, such as reading and math. This report summarizes the 2019-2020 evaluation procedures 


and results for Virginia 21st CCLC programs. 


The Evaluation Design 

The purpose of the evaluation was to determine whether the 21st CCLC programs were 
addressing the statewide program objectives: (1) improving student academic achievement in 
reading; (2) improving student academic achievement in mathematics; and (3) providing 
opportunities for family engagement. While not one of the statewide objectives, an analysis of 
school-day attendance was also included. 

Data were analyzed from the following sources: (a) 21st CCLC and school-day 
attendance for all student participants, (b) the online Annual Local Evaluation Survey (ALERT), 
(c) the Virginia Annual Performance Report (VAPR), and (d) the Teacher Survey. 


COVID-19 

In early 2020, The World Health Organization declared COVID-19 a global pandemic. In 
response, the government ordered school building closures across the country, including in 
Virginia+, ending all in-person learning. The pandemic created mass disruption of the 218 CCLC 
programs as they switched to virtual learning from home. Due to school closures, data collection 


for the 2019-2020 evaluation was affected, and those effects will be noted throughout the report. 


Conclusions 


The conclusions of the analyses are summarized below by evaluation question. 


What is the nature of the Virginia 21st CCLC grant program and level of participation by 
students? 
A wide variety of activities were offered by 218'CCLC centers during the summer, fall, 


and spring. Homework help, STEM, literacy, and tutoring were the activity types provided most 
1 https://www.doe.virginia gov/support/health_medical/office/covid -1 9-faq.shtml#assess 
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often. Academic support, enrichment, and use of small groups were common practices used to 
improve student academic achievement. Providing students a choice in the selection of activities 
and rotating students through various activities were often-mentioned practices that enhanced 
enrichment opportunities. 

Most 218'CCLC staff were paid school day teachers. Grantees expected all staff and 
volunteers to stress the importance of good character and citizenship and interact positively with 
students during both activities and lessons. Grantees also stressed that frequent, consistent, 
and/or continuous communication with their community partners be maintained to keep them 
engaged in the program. 

The majority of the students attended the program less than 30 days and were in grades 
three through eight. Most student were either White, Black, or Hispanic. Almost two-thirds 
qualified for free/reduced price lunch. School-day teachers reported that the majority of 
substantially served 21st CCLC students improved in behavior and homework participation over 


the school year. 


To what degree did centers meet Virginia’s objectives for the program? 

Without student achievement data from 2019-2020 available, it is difficult to determine if 
the first two objectives were met (improve student academic achievement in reading and math). 
Based on the VAPR, about two-thirds of students who were categorized as “needs to improve” in 
the first quarter improved in both reading and math by the third quarter. Also, those who 
attended the program longer had a slightly higher chance of improving. 

Although centers provided opportunities for family engagement (Objective 3) there was 
low parent participation in the 218'CCLC programs. For every 12 students who participated in 
the program, only one parent participated. Only half of the grantees reported meeting the family 
engagement objective they set in the original grant application, with most providing “Interactive 
Family Literacy” type-programs. The most heavily emphasized promising practice for family 
engagement was outreach, communication, and relationship building with families. Grantees 
repeatedly stressed the importance of using a variety of communication methods with parents, 
noting “consistent parent communication” was key. Incentives for parent engagement and 


participation were also seen as important part of family engagement. 
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What is the impact of 21st CCLC program participation on students’ school-day attendance? 

Participation in 218'CCLC programs had a statistically significant small, but positive 
impact on students 2019-2020 school-day attendance rate, compared to control students. On 
average, students who participated 218'CCLC programs attended close to one day more than 
control students. There was also a statistically significant positive impact on the 2019-2020 
school-day attendance rate foreach subgroup (based on Special Education (SPED) status, 
Limited English Proficient (LEP) status, Economically Disadvantaged (ED) status, gender 
(female), and minority (i.e., non-white) status). Based on the effect size, 218‘ CCLC LEP 
students had the largest positive impact compared to control students. On average, LEP students 
who participated in 218'CCLC programs attended almost one day more than LEP control 
students. 

Within both years examined (2018-2019 and 2019-20), an increase in number of days of 
218'CCLC participation also had statistically significant positive impacts on attendance rates. 
The lowest 218'CCLC participation group (1-29 days) had statistically significantly lower 
attendance rates compared to the other three groups (30-59 days, 60-89 days, and 90+ days) in 
both years for every subgroup with the exception of SPED in 2018-2019, where it was 
Statistically significantly lower than the two highest participation groups. The two highest 
participation groups (60-89 days, and 90+ days) had statistically significantly higher attendance 
rates compared to the two lowest participation groups (1-29 days and 30-59 days) overall, and 
for the female, ED, and minority subgroups. Finally, the highest participation group (90+ days) 
had statistically significantly higher attendance than the next highest participation group (60-89 
days) for the SPED and ED subgroups. 

Between years, 218'CCLC participants in the lowest participation group (1-29 days) were 
the only group to have statistically significant decreases in school-day attendance rates, both 
overall as well as foreach subgroup. The 30-59 days participation group for LEP, and the 60-89 
days participation group for ED and Minority were the only groups to have statistically 
significant increases between years. 

For the 2018-2019 school year, overall, students in the highest 218'CCLC participation 
group (90+ days) on average, had an attendance rate that was approximately 1.5 percentage 
points higher compared to students in the lowest 218'CCLC attendance group (1-29 days). 


Furthermore, this difference increased slightly to almost 2 percentage points in 2019-2020. The 
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largest statistically significant differences between 215'CCLC participation groups were between 
the lowest and highest participation groups in the 2019-2020 school year for the SPED and LEP 
subgroups. For these two subgroups, students who participated in 218'CCLC programs for 90+ 
days had on average, an attendance rate that was 2.5 percentage points higher compared to 
students who participated between 1-29 days. 

Readers should note that the 1-29 days participation group (the group with the least 
number of days attended) was much larger than the other participation groups, making up 
approximately two-thirds to over 80% of the overall samples in both years. Conversely, the 90+ 
days participation group (the group with the most days of attendance) made up approximately 
only 2%-3% of the overall sample. Therefore, comparisons between participation groups should 
be treated with caution due to the large differences in sample sizes. In addition, due to the fact 
that schools closed in March 2020 due to the COVID-19 pandemic, it is not possible to know if 


attendance data for the year would have been different had it been a normal, full school year. 


What promising practices regarding the achievement of required objectives were identified by 
centers? 

Among comments submitted by grantees across the six subjects (math and 
reading/language arts; parent education; character education; enrichment opportunities; and 
community partnerships), the most heavily emphasized “‘promising practices” addressed three 
broad areas. First and most prominently were practices that supported the students. These can be 
broken into three types: Support for academic performance, enrichment activities, and use of 
small groups. The second broad group of practices encompassed family engagement through 
outreach, communication, and relationship building. Finally, there were practices such as 
frequent, consistent, and/or continuous communication aimed at improving community 
partnerships. 

Grantees were also asked for recommendations to improve the program in the future. 
Most were for practices already mentioned in the six subjects listed in the previous paragraph. 
Among ideas not already emphasized were recommended practices under the broad themes of (a) 
family communication and engagement, (b) program structure due to COVID 19, and (c) staff 


retention. 
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Introduction 


This report summarizes the 2019-2020 evaluation procedures and results for Virginia 
21st Century Community Learning Centers (21st CCLC) programs. The mixed-methods 
evaluation utilized perceptual as well as program and school-day attendance data from study 
participants. 

The Center for Research in Educational Policy (CREP), Virginia’s 218'CCLC evaluator, 
is a State of Tennessee Center of Excellence and is located at The University of Memphis. 
CREP’s mission is to implement a research agenda associated with educational policies and 
practices in preK-16 schools, and to provide a knowledge base for use by educational 
practitioners and policymakers. Since 1989, the Center has served as a mechanism for mobilizing 
community and university resources to address educational problems and to meet the 
University’s commitment to primary, secondary, and higher education institutions. Functioning 
as part of the College of Education, the Center seeks to accomplish its mission through a series 
of investigations conducted by Center faculty, staff, and associates, College and University 


faculty, and graduate students. 


Background and Program Description 

The 21st Century Community Learning Centers (CCLC) program was established by 
Congress as Title X, Part I, of the Elementary and Secondary Education Act (ESEA). It was 
reauthorized by Congress under the Every Student Succeeds Act of 2015 (ESSA). The purposes 
of the 21st CCLC program are as follows: 

e To provide academic enrichment opportunities outside of the regular school day to help 
students, particularly students who attend high-poverty and low-performing schools, meet 
state and local performance standards in core academic subjects. 

e To offer students a broad array of services, programs, and activities to complement 
academics, such as drug and violence prevention; counseling programs; art, music, and 
recreation programs; technology education; and character education. 

e To offer families of students served by community learning centers opportunities for literacy 


and related educational development. 
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21% Century Community Learning Centers in Virginia 

Every year, applicants apply for competitive 21st CCLC grant funds through the Virginia 
Department of Education (VDOE). Those awarded the 218'CCLC grant money are part of the 
three-year grant cycle, and are required by VDOE to participate in data collection, monitoring, 
and evaluation. Programs provide students with academic and enrichment opportunities before 
and/or after school, and some offer programs during the summer as well. Collaboration with 


parents of 218'CCLC students and community partners is also expected within these programs. 


Evaluation Objectives and Questions 

States, as the recipients of 218'CCLC funds, are responsible for providing comprehensive 
evaluations of their programs. CREP was contracted by the VDOE to conduct a statewide 
evaluation and to assess the extent to which local grantees met the following defined 
programmatic objectives: 

Objective 1: Improve student academic achievement in reading. 

Objective 2: Improve student academic achievement in mathematics. 


Objective 3: Provide opportunities for family engagement. 


To address the 218'CCLC objectives, CREP’s evaluation is structured around the following 
questions: 

1. What is the nature of the Virginia 21‘'CCLC grant program and level of participation by 
students? 

2. To what degree did centers meet Virginia’s objectives for the program? 

3. What is the impact of 218'CCLC program participation on students’ school-day 
attendance? 

4. What “promising practices” regarding the achievement of required objectives were 


identified? 


COVID-19 
In early 2020, The World Health Organization declared COVID-19 a global pandemic. In 


response, the government ordered school building closures across the country, including in 
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Virginia”, ending all in-person learning. The pandemic created mass disruption of the 218tCCLC 
programs as they switched to virtual learning from home. Due to school closures, data collection 


for the 2019-2020 evaluation was affected, and those effects will be noted throughout the report. 


? https://www.doe.virginia. gov/support/health_medical/office/covid-1 9-faq.shtml#assess 
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Methods 


Participants 

For the 2019-2020 evaluation, which includes grant recipients within a three-year grant 
cycle (Cohorts 15R, 16, 17, & 18), 115 programs served 132 schools. Most centers served one 
school; however, 11 centers served two, three, or four schools. The population consisted of (a) 
grantees and/or site coordinators, (b) school-day teachers and administrators from participating 
schools, (c) after-school teachers, (d) volunteers, (e) student participants, and (f) the 
parents/guardians of student participants. The study population, along with others associated 
with the program, is discussed in detail in the report section Center and Participant 


Characteristics found on page 17. 


Instrumentation 

During the 2019-2020 school year, data were collected from three main sources: (a) The 
online Annual Local Evaluation Survey (ALERT), (b) the Virginia Annual Performance Report 
(VAPR), and (c) the Teacher Survey. The Student Perceptual Survey was not administered this 
year due to school closures. 

Annual Local Evaluation Report Template (ALERT). The ALERT is an online tool a 
grantee is required to submit for each center after a full year of program implementation. 
Grantees with multiple sites serving different students at each site must complete a separate 
ALERT foreach site. The self-reporting tool was opened for approximately two months during 
the summer of 2020. Its purpose is to gather data regarding measurable objectives, activities, and 
outcomes. Grantees were also asked to describe the “promising practices” they found most 
helpful, and to provide recommendations for improvements to the program. It is important to 
note that grantees reported their individual levels of success in meeting objectives based on their 
own pre-determined criteria (vs. an objective measure). 

Virginia Annual Performance Report (VAPR). The VAPR is data the grantees submit 
to VDOE through a web-based system. The data collected is required by the United States 
Department of Education for annual reporting on the progress of the state’s 218‘ CCLC programs 
and is based on the Government Performance and Results Act (GPRA) measures established by 
congress. For the 2019-2020 year, and for the years going forward, grantees report VAPR data in 
Cayen Systems, a Web-based system designed for Virginia’s 218'CCLC data collection. 
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Teacher Survey. The Teacher Survey was designed to collect information from the 
regular school-day teacher about changes in behavior and homework completion for each 21st 
CCLC student. These data were also reported online in Cayen Systems. 

Student Perceptual Survey. The Student Perceptual Survey was developed to give 
students the opportunity to anonymously provide their perceptions of the 21st CCLC program 
and a means to report benefits they attribute to their program attendance. Students in grades 3-12 
who participated in the program 30 or more days (i.e., were substantially served) were asked to 


participate in the survey. 


Analyses 

Data were analyzed from four main sources: (a) 21st CCLC and school-day attendance 
data on all student participants, (b) the online Annual Local Evaluation Survey (ALERT), (c) the 
Virginia Annual Performance Report (VAPR) Survey, and (d) the Teacher Survey. These 
sources are summarized by evaluation question in Table | below, followed by a detailed 


description of the statistical analyses used to analyze school-day attendance. 


Table 1. Summary of Instruments and Data Sources by Evaluation Question 
Evaluation Question Data Sources 


ALERT 

Virginia Annual Performance Report (V APR) 
Teacher Survey 

ALERT 

Virginia Annual Performance Report (VAPR) 
Teacher Survey 


What is the nature of the 21st CCLC programs 
and level of participation by students? 


To what degree did centers meet Virginia’s 
objectives for the program? 


What is the impact of 21st CCLC program 
participation on students’ school-day 
attendance? 


Teacher Survey 
21st CCLC and school-day attendance data 


What “promising practices” regarding the 
achievement of required objectives were 
identified by centers? 


Analysis of School-day Attendance. In the past, analyses of data from the Standards of 
Leaming (SOL) and Virginia Alternative Assessment Program (VAAP) were conducted to 
determine the extent to which local grantees met programmatic objectives related to improved 
academic achievement in reading and mathematics. However, for the 2019-2020 school year, 


SOL and VAAP test data were not available due to the cancellation of Commonwealth 
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assessments resulting from COVID-19 pandemic-related school closures. In addition, center- 
level data were not available from the current Cayen Systems. As a result, an analysis of two 
years of available school-day attendance was conducted instead. The attendance analysis was 
structured around the following question: What is the impact of 21°" CCLC program 
participation on students’ school-day attendance? 

The VDOE provided CREP the following data by student for a two-year analysis of 
school-day attendance: 

e School name 

e Research ID 

e Grade level 

e Gender 

e Race 

e Limited English proficiency status (LEP) 

e Disability status (SPED) 

e Economically Disadvantaged status (ED) 

e Number of days of participation in 21*'CCLC 

e Number of school days in session (2018-2019 and 2019-2020) 

e Number of days present (2018-2019 and 2019-2020) 

e Number of days absent (2018-2019 and 2019-2020) 


The 2019-2020 population of students for the school-day attendance analysis consisted of 
students in grades 3-12 enrolled in schools served by a 21st CCLC grantee in cohorts 15R, 16, 
17, or 18 (VN = 62,554). 

Two sets of school-day attendance analyses were conducted. The first set compared the 
2019-2020 school-day attendance rate (i.e., percentage of days present) between 21st CCLC 
participants and matched nonparticipant groups overall, as well as by five subgroups, based on 
Special Education (SPED) status, Limited English Proficient (LEP) status, Economically 
Disadvantaged (ED) status, gender (female), and minority (i.e., non-white) status for the 2019- 
2020 school year, after controlling for prior-year attendance. Only students who were 


“substantially served” by a center (1.e., attended 30 or more days) were considered 21st CCLC 
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program participants (n = 6,113). Students who were eligible to attend, but had zero days of 
attendance, were considered control students (n = 56,441). 

To be included in the analysis, students had to have attendance data from both the 2018 - 
2019 and 2019-2020 school years. In addition, the sum of days present and days absent had to 
equal the total number of school days in session for the year, otherwise the record was deleted 
(Available sample: CCLC n = 4,961, control n = 49,623). Control students were matched with 
21st CCLC students based on their 2018-2019 calculated school-day attendance rate (days 
present/school days in session). In addition, students were matched on grade level, gender, LEP 
status, SPED status, ED status, and minority status. The final matched file consisted of 4,961 
21st CCLC and 4,961 matched control students. 

The second set of school-day attendance analyses compared four CCLC participation 
groups (1-29 days, 30-59 days, 60-80 days, and 90+ days) for 21st CCLC students only for the 
2018-2019 and 2019-2020 school years, as well as the subgroups SPED, LEP, ED, female, and 
minority. To be included in the analysis, students had to have attendance data from both the 
2018-2019 and 2019-2020 school years. Also, the sum of days present and days absent had to 
equal the total number of school days in session for the year, otherwise the record was deleted. 
This file consisted of 15,000 21st CCLC students with varying days of participation. 

When students are nested within schools (or centers in this case), Hierarchical Linear 
Models (HLM) is the preferred method of analysis, as it takes the nested structure of the data into 
account. However, initial results from HLM indicated less than one percent of the total variance 
in 2019-2020 attendance outcomes was between schools, thereby reducing the need fora 
hierarchical analysis of the data. Therefore, all outcomes were analyzed using Generalized 
Linear Models (GLM), as binary logistic regressions (i.e., dichotomous outcome), with the 
outcome variable being the number of events occurring (number of days present) in a set of trials 
(number of days in session). This allows all students to be compared using the same scale (i.e., 
percentage of days present), regardless of any variation in the number of days each school was in 
session. 

In addition to testing outcomes for statistical significance, effect sizes, percentile ranks, 
and improvement indices associated with the treatment effect were calculated to aid in the 
interpretation of findings. As an indicator of the impact or “practical significance” of the 


treatment, the effect size (typically calculated as Hedges’ g) is a descriptive statistic that 
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indicates the magnitude of the difference (in standard deviation units) between two measures. 
The What Works Clearinghouse (WWC) (2020a) has adopted the Cox Index effect size (CIES) 
for dichotomous outcomes, which provides an effect size similar to Hedges’ g. In addition, the 
WWC translates the Hedges’ g and Cox Index effect size into an “improvement index”, which 
can be interpreted as, “the expected change in percentile rank for an average comparison group 
student if that student had received the intervention.” 

A further consideration is that the attendance findings can only be used to evaluate the 
performance of all centers in Virginia as a group, not the performance of any specific center, as 
results were aggregated across all centers rather than evaluated center-by-center. Also, because 
of the varied impact of COVID-19 on schools throughout Virginia, methods for recording and 
collecting attendance data for 2019-2020 could have varied considerably by school. 
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Center and Participant Characteristics 


Center and participant characteristics are reported by grantees in Cayen Systems: 107 
centers completed a report for summer 2019, and 124 centers completed a report for the 2019- 
2020 regular school year. Specific staff, student, family member, and activities are described 


below. 


Staff 

As seen in Table 2, more than 80% of the staff were paid, and less than 20% were 
volunteers. Fall had the highest percentage of volunteers (18%) and spring had the highest 
percentage of paid staff (87%). 


Table 2: Paid and Volunteer Staff 


Total # of staff Numberof %ofPaid Number of % of Ree 
ate sae Paid Staff Staff Volunteers Volunteers Nera 


School-day teachers were the most common staff member working in the 218' CCLC 


programs during all three semesters (Table 3). Non-teaching staff and college students were the 
next most common type of staff. Approximately one percent of parents helped in the centers 


during the regular school year, and 5% helped during the summer. 


Table 3: Type of Staff 
Non- NYo store) Nil exereya} 


Stee nae - S sey Other | Teaching | Parent Day tracted 
Staff Teacher Staff 


Admin 
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Students 
Summer 2019 Program. A total of 5,749 students in Pre-K through 12" grade attended 


21st CCLC summer programs in 2019. The majority (98%) of 218‘ CCLC student participants 
attended 1-29 days (See Table 4). Most students were in grades three through eight (77%), with 
grade six (16%) having the largest number of student participants, followed closely by grade 
seven (14%). Pre-kindergarten and high school students (9-12) were the least likely to participate 
because the majority of the grants received and awarded are for upper elementary and middle 


school grades (See Table 5) 


Table 4. Summer Student Attendance by Days Served 
| Attendance Day Category Number Percentage 


5655 
30-59 days 


60-89 days 7 0% 
foordays Ta 
TOTAL 5,749 100% 


Table 5. Summer Student Attendance by Grade Level 
| Grade Level | Number Percentage 


Pre-kindergarten ee 


Kindergarten 


1st grade 
2nd grade 
3rd grade 
4th grade 


Sth grade 


6th grade 
7th grade 


8th grade 
9th grade 
10th grade 
11th grade 
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Demographics. The summer demographic information collected in the VAPR reflect 
that nearly half of the student participants were male, and half were female (Table 6). The ethnic 
groups with the highest percentage of students served were Black (49%), White (34%), and 
Hispanic (10%). Approximately 6% of students had limited English proficiency, 72% were 


reported as eligible for free/reduced-price lunch, and 12% had a special need. 


Table 6. Summer Student Demographics 
Student Demographics | Number Percentage 


Gender 


Male 49% 
Female 49% 
Unknown 2% 

TOTAL 100% 


Ethnicity 


American Indian 


Hispanic 547 


a 


Regular School Year Program. Grantees reported that 20,863 students were in 
attendance at least one or more days during the 2019-2020 regular school year (Table 7). Of that, 


38% (n = 7,942) were substantially served (i.e., attended 30 or more days). 
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Table 7. Regular School Year Student Attendance by Days Served 
| Attendance Day Category Number Percentage 


60-89 days 2,382 
90+ days 1,011 
TOTAL 20,863 100% 


Similar to the summer program, the majority of students served were in grades three 
through eight (77%) (Table 8). Grade seven had the largest number of student participants 
(20%), followed closely by grade eight (18%). As noted previously, pre-kindergarten and high 
school students (9-12) were the least likely to participate because the majority of the grants 


received and awarded are for upper elementary and middle school grades. 


Table 8. Regular School Year Student Attendance by Grade Level 
| Grade Level | Number | Percentage 


Kindergarten 509 2% 
1st grade 710 3% 


9th grade 629 3% 
10th grade 729 3% 


11th grade 509 2% 
12th grade 472 2% 


TOTAL 20,863 100% 
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Demographics. The regular school year demographic information reflect that 
approximately half of the students served were female, and half were male (Table 9). The ethnic 
groups with the highest percentage were White (38%), Black (32%), and Hispanic (20%). 
Approximately 10% of students had limited English proficiency, 62% qualified for free/reduced 
price lunch, and 12% had a special need. While many of the regular school year percentages are 
similar to the summer program, the exceptions are that the summer program served a higher 
percentage of Blacks than Whites, as well as a higher percentage of students that qualify for free 
and reduced lunch. 

Compared to the state. When comparing the 21st CCLC student population to all the 
students served throughout the Commonwealth of Virginia for the 2019-2020 school year, the 
21st CCLC student population was representative of the Commonwealth in some particulars, but 
not others (See Table 9). The 218'CCLC programs served a higher percentage of Black students 
and students who qualify for free/reduced price lunch, and a lower percentage of White students 


than the state overall (Virginia Department of Education, 2019-2020). 


49% 


x 


Special Needs 2,423 13% 
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Family Members 

The 21st CCLC programs also served family members of 21st CCLC students. Grantees 
reported a total of 734 family members who attended 21st CCLC programs during the summer of 
2019, and 1,823 family members in attendance during the regular school year (2019-2020). 


Activities 

A wide variety of activities were offered to students by 218'CCLC centers, including arts 
& music, college & career readiness, community/service learning, counseling programs, drug 
prevention, English language learner support, entrepreneurship, homework help, literacy, 
mentoring, physical activity, STEM, truancy prevention, tutoring, violence prevention, and youth 
leadership. 

As seen in Table 10, homework help, STEM, literacy, and tutoring were the activity types 
provided most often during the 2019-2020 school year. Drug prevention, violence prevention, 
English language learner support, truancy prevention, and entrepreneurship were the activities 


provided the least. 


Table 10. Activities Offered by Semester 
Activity Fall 2019 | Spring 2020 


College & Career Readiness 
Community / Service Learning 
English Language Learners Support 
Literacy 
Mentoring 
Physical Activity 

TEM 
Truancy Prevention 
Violence Prevention 
Youth Leadership 
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Results 


Grantees are required to address three objectives: 1) Improve student achievement in 
reading/language arts; 2) Improve student achievement in mathematics; and 3) Provide 
opportunities for family engagement. The extent to which centers met these objectives is 
presented below. While not one of the statewide objectives, an analysis of two years of available 
school-day attendance was conducted, followed by aggregate results of the Promising Practices 


and the Teacher Survey. 


Objective 1 & 2 - Improve student achievement in reading/language arts & Improve 
student achievement in mathematics 

As previously stated, an analysis of data from the Standards of Learning (SOL) and 
Virginia Alternative Assessment Program (VAAP) were conducted in past years to determine 
program impacts on student achievement in reading and math. However, for the 2019-2020 
school year, SOL and VAAP test data were not available due to the cancellation of 
Commonwealth assessments resulting from COVID-19 pandemic-related school closures. 
Therefore, only results from the VAPR were analyzed to address the first and second objective. 

VAPR: During the first quarter of the regular school year students are categorized as 
“needs to improve” in both reading and math if they have a grade of “C” or less. Third quarter 
grades are then reviewed to see if those “needs to improve” students who attended 30 days or 
more improved by one letter grade, or for grades that are reported by a percentage, an increase of 
five percentage point. Grantees from 125 centers entered grade outcome data for their 218'CCLC 
students. Those outcomes are reported below in Table 11. 

Approximately two-thirds of 218'CCLC students who were categorized as “needs to 
improve” actually improved in both reading and math by the third quarter. Students who 
attended the program 90 days or more usually had a higher chance of improving their grades than 


student who attended either 30-59 days or 60-89 days. 
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Table 11. Student Grade Outcome Data 
Needs 

Needs to a 6 Needs to fe ee oe: oA 

Gaile tears Improved To Improve To to Improved To 


; ie improve ‘ nam ee : 
ee Nyelo(ce 30-59 30-59 Impro 60-89 d 60-89 Improv improve 90+ Impr 
DEW 


Days ved Day IDE WS ed 90+ Days oved 
IBEW AS 


Objective 3 - Provide opportunities for family engagement. 

According to the VAPR, atotal of 734 family members attended 21st CCLC programs 
during the summer of 2019, while 1,823 family members attended during the regular school year 
(2019-2020). 

Inthe ALERT, grantees were asked to report (a) the program’s measurable objective for 
family engagement, (b) the outcomes for that objective, and (c) if they met the objective. About 
half reported the family engagement objective was met (See Table 12). 


Table 12. Family engagement objective data, 2019-2020 


: nate ; Ifo) 
Did you meet your objective for... No Yes Reponee Responses 


When asked about types of family engagement activities the program provided, 


“Interactive Family Literacy” was the most common type provided (Table 13). 


Table 13. Type of family engagement, 2019-2020 
Identify the types of family engagement activities the program 


arovicede (Checkalldnegayea Number Percentage Responses 


Interactive Family Literacy 


Economic Self-sufficiency Training 


neering) | 
Primary Teacher Training 
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School-day Attendance Analysis 

Matched 21st CCLC and Control Groups. 21st CCLC students in the female and ED 
subgroups had statistically significantly lower attendance rates for the baseline 2018-2019 school 
year compared to matched control students. There were no other statistically significant 
differences between the 21st CCLC and matched control groups for the 2018-2019 baseline 
attendance rates (see Table 14). Although not statistically significant, CCLC mean school-day 
attendance rates were slightly lower than controls overall and for the subgroup SPED. However, 
while there were statistically significant differences, baseline equivalence between the CCLC 
and control groups was established as none of the CIES effect sizes were above the WWC 


(2020b) established threshold (i.e., [Baseline ES| > 0.25). 


Table 14. Mean Baseline 2018-2019 School-Day Attendance Rates for 21st CCLC and Control 
Groups 


CCLC Control | 
Attendance Attendance Odds Percentile Improvement 
Subgroup N Rate N Rate SD _ Coefficient | Ratio Dp CIES Rank is 


pos Tusa oes uals [ona [sal oor om Pons Poof ao [a 
| Femates | 2577| 96.1 | 3.9| 2521] 963 [36] -o05 | 095 | <oorm|oos | 49 | 1 | 


ruse | as | 967 [36] «26 | 67 [34] 000 —[ 100 [096s [000 | —s0 | 0 
jsrep | su | 95.3 | 49| sar | 954 [46] -002 | 098 | 0.209 | oor | 50 | 0 | 
pep [342[ 957 [433440] 958 [41] 003 | 098 | 07 |-oo1 | so | oi 
| Minority | 3393] 96.2 [41] 3303] 96.2 [40] -oo1 | 099 | 0553 | oo1| 50 | 0 | 


*Statistically significant at p < 0.05. 


When examining the 218'CCLC participant and matched control groups in grades three 
through twelve overall, after controlling for prior year school-day attendance, participation in 
218'CCLC programs (“Yes” or “No”) had a statistically significant positive effect on 
participants’ 2019-2020 school-day attendance (see Table 15), with the effect size (CIES = 0.12) 
indicating that 55% of 218'CCLC students had higher attendance than the average control group 
student. 218'CCLC students attended slightly less than one day more on average compared to 
controls. 

In addition, there were statistically significant positive differences in the 2019-2020 
school-day attendance favoring 218'CCLC participants for all subgroups: (a) female (CIES = 
0.12), (b) LEP (CIES = 0.15), (c) SPED (CIES = 0.13), (d) ED (CIES = 0.11), and (e) minority 
(CIES = 0.13). Based on the effect size, the LEP subgroup had the largest statistically significant 
positive difference, with 56% of 218'CCLC LEP students having higher attendance than the 
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average control LEP student. Overall, 21‘'CCLC LEP students attended nearly one more day on 


average compared to LEP control students. 


Table 15. Mean 2019-2020 School-Day Attendance Rates for 21st CCLC and Control Groups 


CCLC Control | 
Attendance Attendance Odds Percentile | Improvement 
Subgroup N Rate RY) N Rate SD | Coefficient Ratio Pp CIES Rank Index 


fuse | 626 [972 [s0[ 6 | 965 [3s[ 025 [128 [soo [ ois [se [6 | 
-sesp_| sat {_oss_{s] sat | os {oa | _o22 {ize | coors} ots | ss _{ _s__ 
pe__fuet|_shy__ ada, _v8)_Se} au fas 54 


‘aaa [sass] ssa [aa] s300] ase [sa] am [am [ame oa | ss | 5 


*Statistically significant at p < 0, 05. 


21% CCLC Only Students. 218‘ CCLC students’ program participation was divided into 
four groups: (a) 1-29 days, (b) 30-59 days, (c) 60-80 days, and (d) 90+ days. The 2018-2019 
baseline and 2019-2020 outcome school-day attendance (number of days present occurring in a 
set of number of days in session) were treated as a repeated-measures variable (Time). The GLM 
repeated measures analysis produced three outcome effects: (a) 218'CCLC participation group 
main effect, (b) Time main effect, and (c) Group and Time interaction effect. An interaction 
effect (Group * Time) occurs when the effect of the Group variable depends on the value of the 
Time variable. When this interaction is statistically significant, independent interpretation of the 
Group or Time main effect is not possible. 

Table 16 through Table 21 display the results of the GLM repeated measures analyses 
and school-day attendance rate means by participation group and time for all 218'CCLC 


students, as well as each subgroup. 
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Figure 1 through Figure 6 are graphic representations of the participation group and time 
interaction effects. Overall, and for each subgroup, the interaction effect forGroup*Time was 
statistically significant (p < 0.05), indicating that the effects for 21’'CCLC participation group 
were dependent on time. 

Overall, 218‘ CCLC students in the lowest participation group (1-29 days) had a 
statistically significant decrease in school-day attendance from 95.4% in 2018-19 to 94.9% in 
2019-2020 (p < 0.01), while the three highest participation groups had small, non-statistically 


significant increases in school-day attendance (see Table 16 and 
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Figure 1). However, the size of the decrease for the lowest participation group (-0.5 
percentage points), was small. 

During the 2018-2019 and 2019-2020 school years, the three highest participation groups 
had statistically significantly higher school-day attendance compared to the lowest (1-29 days) 
participation group, with the differences ranging from 0.5 to 1.9 percentage points. Additionally, 
the two highest participation groups had statistically significantly higher school-day attendance 
compared to the 30-59 days participation group in both 2018-2019 and 2019-2020, with the 


differences ranging from 0.4 to 0.9 percentage points. 


Table 16. All 21st CCLC Students - Repeated Measures Results for = Da essere 
| Effect Wald Chi-Square 


CCLC Participation Mean School-Day 


Group Attendance | Std. Dev. Ay 


B C 1D) E 
SL 
2018-2019 
Attendance Rate 
1-29 days | 9490 | .50_| 10,039 | 66.9_| 
2019-2020 


Attendance Rate 60-89 days° 96.52% 1,631 
96.87 


*Statistically significant at p < .05. 

Superscripts in Column C indicate statistically significant differences within year(e.g., 
B was statistically significantly different from >-* and¢in Column B). 

Meansin red indicate a statistically significant decrease between years. 

Meansin green indicate a statistically significant increase between years. 


bed indicates that * in Column 
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Figure 1. Interaction Effects for All 21st CCLC Students 


CCLC 
97.00 Participation 
——_—_—_——— Group 
96.87 
Beye =" 1-29 days(a) 
m= 30-59 days(b) 
e wm" 60-89 days(c) 
ia 96.50 mmr 90+ days(d) 
o 
= 96.31 
o 
c 
o 
TC 96.00 
s —— ooo 
= 95.89 
c 
o 
o 
= 
95.50 
95.35 
95.00 [94.94] 
2018-2019 2019-2020 
Time 


Female 218'CCLC students in the lowest participation group (1-29 days) had a 
statistically significant decrease in school-day attendance from 95.4% in 2018-19 to 95.0% in 
2019-2020 (p < 0.01), while the three highest participation groups had a small, non-statistically 
significant increase in school-day attendance (see Table 17 and Figure 2). However, the size of 
the decrease for the lowest participation group (0.4 percentage points), was small. 

During the 2018-2019 and 2019-2020 school years, the three highest participation groups 
had statistically significantly higher school-day attendance compared to the lowest participation 
group, with the differences ranging from 0.5 to 2.0 percentage points. Additionally, the two 
highest participation groups had statistically significantly higher school-day attendance 
compared to the 30-59 days participation group in 2018-2019 and 2019-2020, with the 


differences ranging from 0.5 to 1.0 percentage points. 


Table 17. Female 21st CCLC Students - Repeated Measures Results for School-Day Attendance 
| Effect Maeickertesctetc 


29169.64 


df 
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CCLC Participation Mean School-Day KY 7 A Ya N out 
Group 


B Cc D E F 

1-29 days" 95.40% 5,081 

2018-2019 30-59 days" 95.89%! 1,491 
Attendance Rate | 60-89 days° 96.42% 


Attendance Sample 


*Statistically significant at p < .05. 

Superscripts in Column C indicate statistically significant differences within year(e.g., “indicates that * in Column 
B was statistically significantly different from °° and 4in Column B). 

Means in red indicate a statistically significant decrease between years. 

Meansin green indicate a statistically significant increase between years. 


Figure 2. Interaction Effect for Female Students 


CCLC 
Participation 
Group 
=== 1-29 days(a) 
—— 30-59 days(b) 
w= 60-89 days(c) 

mmr" 90+ days(d) 


97.00 


96.50 


96.00 


Mean Attendance Rate 


95.50 


95.00 


2018-2019 2019-2020 


Time 


LEP 21*'CCLC students in the lowest participation group (1-29 days) had a statistically 
significant decrease in school-day attendance from 95.8% in 2018-2019 to 95.4% in 2019-2020 
(p < 0.01), while the 30-59 days participation group had a statistically significant increase in 
school-day attendance from 96.4% in 2018-2019 to 97.1% in 2019-2020 (p <0.01). The 


differences between years (-0.4 and 0.7 percentage points respectively), however, were small. 
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For the 2018-2019 school year, the three highest participation groups had statistically 
significantly higher school-day attendance compared to the lowest participation group, with the 
differences ranging from 0.6 to 1.5 percentage points. Additionally, the 60-89 days participation 
group had statistically significantly higher (0.5 percentage points) school-day attendance 
compared to the 30-59 days participation group. For the 2019-2020 school year, the three 
highest participation groups had statistically significantly higher school-day attendance 
compared to the lowest participation group, with the differences ranging from 1.7 to 2.5 
percentage points (see Table 18 and Figure 3). 


Table 18. LEP 21st CCLC Students - Repeated Measures Results for School-Day Attendance 

| Effect Matters Srte df 

Mince ISTP) 0000 

|CCLCParticipationGroup | 100.52 | 30.000" 
[Time —“‘“‘(S*rSC“‘OROO CUT Ct 0095 

|Group*Time | 8.120000 | 3 | 0.000" 


CCLC Participation | Mean School-Day Std. % of 
Attendance Dev. Sample 


Group 


B C 1D) E F 
er 
sentnceRate | 0.s9uuys | soo [359 [ier | 60 
[ s0edays [ona Pa [99 8 
2019-2009 [30-59dayx?) | ora | 3.00_[ 370 
Attendance Rate [60:89 days" | 9727" ‘| -a2a_| 197 
P90 days* | 97.90" | 27 [59 


*Statistically significant at p < .05. 

Superscripts in Column C indicate statistically significant differences within year(e.g., “indicates that * in Column 
B was statistically significantly different from >-* and4in Column B). 

Means in red indicate a statistically significant decrease between years. 

Means in green indicate a statistically significant increase between years. 
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Figure 3. Interaction Effect for LEP Students 
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SPED 21%‘ CCLC students in the lowest participation group had a statistically significant decrease 
in school-day attendance from 94.5% in 2018-19 to 93.9% in 2019-2020 (p < 0.01), while the three 
highest participation groups had a small non-statistically significant increase in school-day 
attendance (see 

Table 19 and Figure 4). The change between years for the lowest participation group was 


small, at -0.6 percentage points. 

For the 2018-2019 school year, the highest participation group (90+ days) had a 
statistically significantly higher school-day attendance compared to the other three participation 
groups, with the differences ranging from 1.3 to 2.2 percentage points. Additionally, the 60-89 
days participation group had had statistically significantly higher school-day attendance 
compared to the 1-29 days participation group (0.9 percentage points). For the 2019-2020 school 
year, the three highest participation groups had statistically significantly higher school-day 
attendance compared to the lowest participation group, with the differences ranging from 1.2 to 
2.8 percentage points. In addition, the highest participation group (90+ days) had statistically 
significantly higher school-day attendance compared to the other three participation groups, 


ranging from 1.0 to 2.8 percentage points. 


Table 19. SPED 21st CCLC Students - Repeated Measures Results for School-Day Attendance 


| Wald Chi-Square 
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CCLC Participation Mean School-Day % of 
Group Attendance Sample 


| Std. Dev. N 


B Cc —) E | F 
1-29 days’ 94.524 1,523 


*Statistically significant at p < .05. 

Superscripts in Column C indicate statistically significant differences within year(e.g., 
B was statistically significantly different from °° and4in Column B). 

Means in red indicate a statistically significant decrease between years. 

Means in green indicated statistically significant increase between years. 


bed indicates that * in Column 


Figure 4. Interaction Effect for SPED Students 
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ED 21%'CCLC students in the lowest participation group (1-29 days) had a statistically 
significant decrease in school-day attendance from 94.9% in 2018-2019 to 94.4% in 2019-2020 
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(p < 0.01) (-0.5 percentage points), while the 60-89 days participation group had a statistically 
significant increase in school-day attendance from 95.9% in 2018-2019 to 96.2% in 2019-2020 
(p = 0.017) (0.3 percentage points). 

For the 2018-2019 as well as the 2019-2020 school years, there were statistically 
significantly differences in school-day attendance between each of the CCLC participation 
groups, with each higher participation group having better attendance then each subsequently 


lower participation group (see Table 20 and Figure 5). 


Table 20. ED 21st CCLC Students - Repeated Measures Results for za Attendance 
| Effect Wald Chi- anes 


nee STOO 
-CCLCParticipationGroup | 223-89 | 3 | 0.000" 


Mean School-Day Std. Dev % of 


CCLC Participation Attendance Sample 


Group 


B C 1D) E F 
2018-2019 2.014 
Attendance Rate 
[sera [ae [a8 [5 
2019-2020 
Attendance Rate 
[90+ days’ | 96.15" [3.7 | 295 | 


*Statistically significant at p < .05. 

Superscripts in Column C indicate statistically significant differences within year(e.g., “indicates that * in Column 
B was statistically significantly different from >-* and¢in Column B). 

Means in red indicate a statistically significant decrease between years. 

Means in green indicate a statistically significant increase between years. 
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Figure 5. Interaction Effect for ED Students 
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For minority 218‘ CCLC students, the lowest participation group (1-29 days) had a 
statistically significant decrease in school-day attendance from 95.4% in 2018-19 to 95.0% in 
2019-2020 (p < 0.01) (-0.4 percentage points), while the 60-89 days participation group had a 
statistically significant increase in school-day attendance from 96.3% in 2019-2019 to 96.7% in 
2019-2020 (p = 0.030) (0.4 percentage points) (see Table 21 and Figure 6). 


During the 2018-2019 and 2019-2020 school years, the three highest participation groups 
had statistically significantly higher school-day attendance compared to the lowest participation 
group, with differences ranging from 0.6 to 2.3 percentage points. Additionally, the two highest 
participation groups had statistically significantly higher school-day attendance compared to 30- 


59 days participation group, with differences ranging from 0.3 to 1.1 percentage points. 
Table 21. Minority 21st CCLC Students - Repeated Measures Results for School-Day Attendance 


| Effect | Wald Chi-Square df P 
29098.67 0.000* 


47.44 


0.381 
0.000* 


CCLC Participation Group 168.43 0.000% 
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Mean School-Day Std. 
Attendance Dev. 


D 


CCLC Participation 
Group 


N 


134 C 


*Statistically significant at p < .05. 

Superscripts in Column C indicate statistically significant differences within year(e.g., “indicates that * in Column 
B was statistically significantly different from >-* and 4in Column B). 

Means in red indicate a statistically significant decrease between years. 

Means in green indicated statistically significant increase between years. 
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Figure 6. Interaction Effect for Minority Students 
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Readers should note that the 1-29 days participation group (the group with the least 
number of days attended) was much larger than the other three participation groups, making up 
approximately two-thirds to over 80% of the overall samples in both years. Conversely, the 90+ 
days participation group (the group with the most days of attendance) made up approximately 


only 2%-3% of the overall sample. Therefore, comparisons between participation groups should 
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be treated with caution due to the large differences in sample sizes. In addition, due to the fact 
that schools closed in March 2020 due to the COVID-19 pandemic, it is not possible to know if 


attendance data for the year would have been different had it been a normal, full school year. 


For context, in comparison to the Commonwealth (Table 22), where the attendance rate across all 
132 school divisions (Commonwealth Total) remained the same across years, with the exception of 
(a) the 1-29 days participation group (which experienced a decline between years across all 
subgroups) and (b) the 90+ days participation group for SPED (which remained the same), all other 
subgroups demonstrated an increase in their attendance rate across years. In addition, compared to 
the Commonwealth ( 

Table 23), with the exception of the 1-29 days participation groups, the vast majority of 


subgroups had attendance rates either (a) at or above the median? for the Commonwealth or (b) 


greater than or equal to the Commonwealth level attendance high for the respective year. 


Table 22. Change in CCLC and Commonwealth Division Attendance Rates across Years: 2018- 
2019 to 2019-2020 


cece Commonwealth 
Group rN II Female | LEP | Ns) D) | 13) | Minority Total 
-0.5 0.4 | -o5 | -0.4 
| 03 (ORM Oe 


| 0.6 _| 


Note: There were 132 school divisions in both years. The source of the Commonwealth level data is the 
Superintendent’s AnnualReport (https://www.doe.virginia.gov/statistics_reports/supts_annual report/index.shtml). 


Values in the cells represent the percentage point change between years. 
Note: Values in highlighted in red are statistically significant declines between years, while values highlighted in 
green are statistically significant increases. Values highlighted in yellow were the same between years. 


Table 23. Comparison of CCLC and Commonwealth Level Division Attendance Rates by Year 
(oe Ke (@oyasvesvoyahyver-l (vey 
Year Group TN Gaia 3-3 100) co 2) s) eas) od =) Dn =) DY | Minority Total Median High Low 


A B c oe ie | el xe H I ieleK 
a ele: 
2018- 
2019 


30-59 days Tee 95.0 | 95.5 RG 
) = 95 95 
60-89 days [RECO 95.4 | 95.9 [eon 
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30-59 days QMO 95.1 | 95.6 |MRaw 


60-89 days [C\iemm ORO mEEOIES 96.2| 96.7 


2019- 
2020 
Pe 969 970 979 96.7 968] 97.3 


3 The medianis the point at which half of the sample falls above, and half of the sample falls below. 
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Note: There were 132 school divisions in both years. The source of the Commonwealth level data is the 


Superintendent’s AnnualReport (https://www.doe.virginia.gov/statistics_reports/supts_annual report/index.shtml), 
Note: Values highlighted in green in are above the Commonwealth level attendance high for the year across all 
school divisions (Column J), while valueshighlighted in yellow are below the median across all school divisions 
(Column I). Numbers in red are statistically significant decreases between years, while numbers in green are 
statistically significant increases between years. 


Promising Practices and Recommendations 

Hundreds of promising practices found to be effective in helping grantees meet their 
objectives were reported in the ALERT. The most frequently mentioned practices are discussed 
below, presented in order of the open-ended questions they address, and organized by themes. 
This is followed by a discussion of the most prominent themes brought out in the 
recommendations. 

1) What activities or promising practices appeared to be most effective in helping to meet 
your subobjectives for improving student academic achievement in reading/language 
arts? 

The top three promising practices for academic achievement in reading/language arts, in 
order were (a) academic support, (b) enrichment, and (c) use of small groups. The most 
commonly mentioned examples of academic support were homework help and 
remediation/tutoring, with many respondents mentioning using them concurrently. Other 
common examples included increased reading opportunities, additional time with teachers, and 
teaching based on grade level or instructional reading level. Suggested ways to increase reading 
opportunities included: utilizing fun activities, ensuring time for independent reading, having a 
book club, extending library hours, or providing guided read ing practice. 

Frequently mentioned examples of enrichment improving reading achievement included 
use of hands-on projects, online programs such as Newsela and CommonLit, activities or 
readings that have real-life applications, and a book club or book of the month. Activities with 
real world applicability included reading recipes and directions for cooking, arts/crafts projects, 
LEGO and K'nexs projects, and board games. 

Approximately a third of respondents explicitly mentioned the use of small groups as 
effective in meeting reading objectives. This included use of small groups for tutoring, reading in 


class, program activities, as well as using one-to-one teacher/student interaction. 
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2) What activities or promising practices appeared to be most effective in helping to meet 
your subobjectives for improving student academic achievement in math? 

Similarly, to Question | regarding promising practices for improving achievement in 
reading/language arts, the top three promising practices for academic achievement in Math were 
academic support, enrichment, and use of small groups. Again, the most commonly mentioned 
examples of academic support included homework help, remediation, and tutoring. Examples of 
support included using personalized software such as myON and IXL, online programs such as 
Contraption Maker, incorporating real world applications rather than memorization, tailored 
instruction that fit the needs of students, as well as SOL based lessons and tools. 

Frequently identified examples of effective and helpful enrichment included DreamBox, 
STEM activities, Kahoots, using manipulatives, and [XL Mathematics. Other examples included 
Math Bingo, Math Seeds, Math Splash, Starfall, Sylvan Learning Center's Ace-It! Program, 
Family Math Nights, Pullout Program, VPM programs, and evidence-based online programs, 
such as Imagine Math and Star Math. Respondents also noted the use of various types of 
apprenticeships or vocational programs such as Emerging Chefs and Outdoor Adventures. 

Mention of the efficacy of small groups for achieving Math objectives paralleled the 
frequency and context of responses about Reading objectives. Approximately a third of 
respondents mentioned the use of small groups as effective, particularly in the context of tutoring 
or in class activities. 

3) What activities or promising practices appeared to be most effective in helping to meet 
your subobjectives for family engagement? 

The most heavily emphasized promising practice for family engagement was outreach, 
communication, and relationship building with families. This was followed in emphasis by a 
variety of family events, and providing workshops, counseling, and other educational sessions 
for parents. 

Grantees repeatedly stressed the importance of using a variety of communication 
methods with parents, noting “consistent parent communication” was key. These included in- 
person communication, flyers, phone calls and robo-calls, newsletters, calendars of events and 
activities, the school marque, e-mail, use of social media, texting, post-card reminders, and 


digital invitations with RSVPs. 
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It was further emphasized that eliciting “parental feedback” was a promising practice. 
Frequently mentioned was the practice of surveying parents to determine their interests, needs, 
when and how to send information, and what they saw as convenient times for activities. 
Accounting for parents’ actual participation in the programs was also seen as a very important 
element for success. 

Some centers identified the use of interpreter services, refreshments, and childcare at 
events to eliminate participation barriers families may face. One center noted, “The ability of 
staff to communicate with families in their home language is also essential. This allows staff to 
assist families in their connection to the school.” 

Among the family educational sessions most prominently mentioned were those 
addressing technology, including computer skills, social networking and social media safety, as 
well as family literacy activities. Programs on nutrition and healthy cooking were also mentioned 
numerous times. Grantees reported various forms of training and support to enhance parenting 
skills as very important as well. 

There were various classes and events noted as particularly effective, such as GED 
preparation classes and college and career information. In addition to events hosting activities 
that were hands-on, activities that engaged the entire family, or facilitated student and parent 
interaction, such as field trips involving children and parents, were frequently reported as very 
engaging and effective. Additionally, events with “fun themes” were noted multiple times as 
being important for increased participation. 

The importance of community partners in providing parent education as well as provid ing 
resources to families was frequently reported. These partners included the local Sheriff's or 
police department, 4-H, YMCA, the Department of Youth and Family Resources, United Way, 
among others. 

Incentives for parent engagement and participation were also seen as a promising 
practice. Provision of food as either meals or snacks was the most commonly mentioned 
incentive. Other incentives included the provision of childcare during events, transportation to 
events, and giveaways of books and other donated items such as door prizes. Additionally, 
basing the timing of activities to coincide with pick-up/end-of-school, multiple offerings of 
activities at different times of day, offerings on the same day as school, and other flexible 


scheduling. 
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4) What activities or promising practices appeared to be most effective in helping to meet 
the program's objective for providing enrichment opportunities? 

Grantees reported a wide variety of options regarding activities, projects, clubs, and 
trainings when asked about enrichment opportunities. Providing students a choice in the 
selection of activities, and rotating students through various activities, were often-mentioned 
practices that enhanced this variety. Hands-on activities including learning instruments, art, 
culinary skills, and gardening were commonly mentioned as well. 

STEM opportunities were the most prominent activity noted. Field trips were also a 
frequently mentioned category of activity. For example, “College and other community-based 
field trips” were mentioned by multiple grantees, along with field trips to local theatres, as well 
as a Civil War Walking Tour. Field trips were also utilized as part of apprenticeship classes 
within several professions. 

Community partnerships were also commonly referenced as helpful with enrichment 
activities, either through guest speakers or specialists, by facilitating field trip venues, or as 
hosting camps. One grantee reported that a partnership “has provided the most effective 
enrichment opportunities”. 

5) What activities or promising practices appeared to be most effective in helping to meet 
the program's objective for providing character education? 

Incorporating character education throughout various programs and activities was the 
primary promising practice reported by grantees in response to this question. Character education 
was seen to improve behavior, class participation, and attendance. Respondents specifically 
described various desired personal traits and skills as being enhanced by character education. 
Most frequently mentioned were teamwork, mentoring, greater emotional regulation and 
understanding, respect for others, and self-respect. 

Enrichment activities and programs were also seen as an important aspect of character 
education. Respondents noted several programs such as Chicas Poderosas, panther pride, various 
sports programs, Girls on the Run, Jr LEGO league, among others. Engaging and interactive 
character education activities was mentioned, with at least one center saying, “no paper-pencil 
activities unless combined with the interactive piece.” 

Mentoring was prominently mentioned as a method of delivering character education, 


followed by integration of character education into programs and direct instruction. In addition to 
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students’ teachers, mentors included individuals from local universities, peer-student mentors, 
high school students, the Department of Youth and Family Resources, Omega U Fraternity, 
Future Leaders of America, and Smart Girls programs. Mentoring groups were noted by some 
grantees as being separated by grade level or gender. Several grantees reported that “gender- 
based mentoring groups” for students was found to be helpful in developing “character and 
leadership skills.” 

Various elements of program structure were reported as promising practices in support of 
character education. The most favorable was ensuring that character education was incorporated 
throughout the program. This included ensuring leadership opportunities, increased group work, 
and having school-wide character expectations. Frequency of character education activities was 
highlighted by several grantees, with some having a monthly character education program or 
activity, whereas another stated, “Character education activities were provided weekly and were 
actively embedded in all facets of the programming”. 

The grantees described collaborations on character education with many community 
partners and vendors as promising practices. Community partners included a county sheriff’s 
office, 4-H, Girl Scouts, YMCA, Boys and Girls clubs, the South-Central Counseling Group, as 
well as others. Services provided by the various partners included mentoring, educational 
sessions for students, assistance with community projects, and alternate dispute resolution, just to 
name a few. 

Staff and volunteers were mentioned by multiple grantees as being essential to effective 
character education. All staff and volunteers were expected to stress the importance of good 
character and citizenship and to interact positively with students during both activities and 
lessons. Some reported that staff utilized “behavior interventions and supports”, ensured students 
had “consistent relationship-building with staff’, and focused “on the students as a whole.” 

6) What activities or promising practices appeared to be most effective in helping to meet 
your subobjectives for improving community partnerships? 

Responses to this question were predominantly about outreach and communication 
among program staff and the partners. The running theme throughout was that programs should 
ensure that frequent, consistent, and/or continuous communication with their community partners 


is maintained. “Communication with community partners was vital to our program”, and 


Virginia 21°‘ CCLC 2019-2020 Evaluation 41 


“communication is key” were just some of the phrases used to describe the perceived 
effectiveness. 

Respondents recommended “constant communication about expectations” before and 
after the grant cycle, as well as consideration of the expectations and needs of grantee and 
community partners. Meeting frequency was also mentioned by multiple grantees, with several 
holding regular and frequent meetings with partners. Multiple centers even chose to assign 
“specific staff members as the liaison with specific community partners.” 

In addition to meetings and having community partners visit sites and attend events, 
specific forms of communication mentioned included email, quarterly newsletters, and telephone 
calls. Communication was often mentioned in conjunction with goals to evaluate progress, 
review programming, develop more community opportunities, discuss logistical needs of 
community partners prior to sessions, and invite partners to engage in family programming 
and/or to visit sites. 

Ensuring that community partners were able to provide input and give feedback regarding 
schedules, programming, and included in broader organizational efforts were also mentioned as a 
promising practice. Moreover, observing community partners in order to “ensure quality 
programming” was another effective promising practice described by a grantee. 

7) What activities or promising practices appeared to be most effective in helping to meet 
the program’s “other” objective? 

When asked about what is helpful to meet “other” objectives, enrichment activities and 
program structure were the two most common themes grantees noted. Multiple grantees 
mentioned Girls of the Run, diversity in activities, career-oriented programming such as college 
and career classes, field trips, and utilizing hands-on activities and projects. Other specifically 
mentioned enrichment activities identified as helpful in achieving their “other” objectives 
include piano keyboarding, mini-economy programming, Jr. Lego league, Drama Matters 
programming, gymnastics, cooking cub, fitness, Science club, STEAM and recreational 
activities, health and fitness, exercise with learning, garden program, Smart Moves, and Money 
Matters. Other grantees noted having dedicated STEM days, targeted campus visits, art-related 
field trips, and morning fitness were beneficial to achieving this objective. One respondent 
highlighted enrichment importance: “Students seemed to attend school more frequently on days 


when enrichment instruction was offered.” 
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Some program structure elements noted by grantees were: “An authentic and welcoming 
environment”, grade-based policy for participation, creating weekly lesson plans, having set 
rotations, incentives for participation such as “Fun Friday” or field trips, frequency of 
programing (such as having STEM programs biweekly), ensuring attendance is highlighted as 
important with students, incorporating student feedback, and free programming and 
transportation. Several respondents commented on continuity between school-day and after- 
school programming through having teachers teaching their own grade-level during the after- 
school program or through consistent communication between teachers and staff. 

8) Provide recommendations that might improve the program in the future. 

Over a hundred responses were submitted, most with multiple recommendations. Many 
of them mentioned themes of family engagement, program structure, or staffing. Some of the 
most common family engagement recommendations included “get a group together to doa 
brainstorm on what family engagement could look like in a different more helpful way to 
families,” ensuring that parent feedback is being incorporated into the program, utilizing a more 
targeted approach in family outreach in lieu of “a wide-net net” which had lower results; and 
increase family trainings. Other respondents mentioned plans to “charge each family with 
attending at least 2 activities” and celebrate participating families. Another respondent wrote 
“Plan, schedule, and organize all of the family engagement events before the school year starts” 
in order to avoid rushing to develop programming and to alleviate stress that occurs when 
planning during the school year. Some recommendations were in response to the ongoing 
pandemic and social distancing requirements: “Identify more opportunities for families to engage 
together at home that are specific, with identifiable goals and/or success;” increasing take home 
resources; and continue to provide resources “related to job opportunities, food assistance, rent 
assistance, social-emotional assistance, and any other needed resources.” 

Many of the program structure recommendations have been noted within the “promising 
practices” above. However, some recommendations not previously discussed included: 
Implementing “virtual learning student packets (due to COVID 19 pandemic)”, creating and 
distributing learning enrichment packets for those students unable to participate in virtual 
learning, changing programming locations due to either attendance or to increase community 
involvement, focus on the data of our students to guide and lead instruction, and providing 


transportation “to students' home stops after school rather than just to designated town stops.” 
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Participants also recommended reducing programming days for middle schoolers from five to 
four to increase retention, “shortening hours to prevent teacher and student burn out,” and 
decreasing programming days to allow for school day closure flexibility. 

Several staff related recommendations have also been noted in previous sections. 
However, significant recommendations include promote and increase staff retention, “retain and 
hire bilingual staff to increase the participation of our Hispanic and English Learner population”, 
encourage homeroom teachers to serve as paid tutors, encourage support staff to help plan, and 
enact character education activities and programming. Several other recommendations worth 
mentioning are to incorporate program guidance counselors, as well as increase professional 


development trainings for all staff. 


Teacher Survey 
Regular school-day teachers were asked to complete one teacher survey for each 215! 

CCLC student who was substantially served (i.e., 30 days or more). Ideally 7,942 surveys would 
have been completed for the 2019-2020 school year, one per 218'CCLC student who attended 30 
days or more. 

e 7,395 surveys were distributed 

e 6,319 surveys were returned 

e Therefore, there was a return rate of 85% 

e And, 80% of substantially served 21*'CCLC students are represented in the 


results from this survey. 


Table 24. Teacher Survey Participation 


Student Grade Number of Surveys Number of Surveys 
Range Distributed Returned 


=e 4432 4,010 


% of Return 


ee 3068 3309 


School-day teachers reported that the majority of substantially served 21st CCLC 
students improved their behavior and homework participation over the school year. Students in 


grades 6-12 who participated 90 or more days were slightly more likely to improve in both 
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homework participation and behavior than 6-12" grade students who participated 30-59 days or 
60-89 days. Students in PreK -5" grade were a little less likely to improve in both homework 
participation and behavior the longer they participated in the program. 


Table 25. 215‘ CCLC Student Outcomes 


Grade Days served by Number of % HW Part. % Behavior 
Category program Students Improved Improved 
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Conclusions 


Overall conclusions are presented below by evaluation question. 
What is the nature of the Virginia 21st CCLC grant program and level of participation by 
students ?# 

A wide variety of activities were offered by 218'CCLC centers during the summer, fall, 
and spring. Homework help, STEM, literacy, and tutoring were the activity types provided most 
often. Academic support, enrichment, and use of small groups were common practices used to 
improve student academic achievement. Providing students a choice in the selection of activities 
and rotating students through various activities were often-mentioned practices that enhanced 
enrichment opportunities. 

Most 218'CCLC staff were paid school day teachers. Grantees expected all staff and 
volunteers to stress the importance of good character and citizenship and interact positively with 
students during both activities and lessons. Grantees also stressed that frequent, consistent, 
and/or continuous communication with their community partners be maintained to keep them 
engaged in the program. 

The majority of the students attended the program less than 30 days and were in grades 
three through eight. Most student were either White, Black, or Hispanic. Almost two-thirds 
qualified for free/reduced price lunch. School-day teachers reported that the majority of 
substantially served 21st CCLC students improved in behavior and homework participation over 


the school year. 


To what degree did centers meet Virginia’s objectives for the program? 

Without student achievement data from 2019-2020 available, it is difficult to determine if 
the first two objectives were met (improve student academic achievement in reading and math). 
However, based on the VAPR, about two-thirds of students who were categorized as “needs to 
improve” in the first quarter improved in both reading and math by the third quarter. Also, those 


who attended the program longer had a slightly higher chance of improving. 


* Due to school closures the student perceptual survey was not administered at the end of the 2019-2020 school year, and 


therefore cannot provide student perceptual data and some insight into the nature of the 21‘ CCLC program. 


Virginia 21°‘ CCLC 2019-2020 Evaluation 46 


Although centers provided opportunities for family engagement (Objective 3) there was 
low parent participation in the 218'CCLC programs. For every 12 students who participated in 
the program, only one parent participated. Only half of the grantees reported meeting the family 
engagement objective they set in the original grant application, with most providing “Interactive 
Family Literacy” type-programs. The most heavily emphasized promising practice for family 
engagement was outreach, communication, and relationship building with families. Grantees 
repeatedly stressed the importance of using a variety of communication methods with parents, 
noting “consistent parent communication” was key. Incentives for parent engagement and 


participation were also seen as important part of family engagement. 


What is the impact of 21st CCLC program participation on students’ school-day attendance? 

Participation in 218'CCLC programs had a statistically significant small, but positive 
impact on students 2019-2020 school-day attendance rate, compared to control students. On 
average, students who participated 218'CCLC programs attended close to one day more than 
control students. There was also a statistically significant positive impact on the 2019-2020 
school-day attendance rate for each subgroup. Based on the effect size, 218'CCLC LEP students 
had the largest positive impact compared to control students. On average, LEP students who 
participated in 218'CCLC programs attended almost one day more than LEP control students. 

Within both years, an increase in number of days of 21’'CCLC participation also had 
statistically significant positive impacts on attendance rates. The lowest 218'CCLC participation 
group (1-29 days) had a statistically significantly lower attendance rate compared to the other 
three groups (30-59 days, 60-89 days, and 90+ days) in both years for every subgroup, with the 
exception of SPED in 2018-2019, where it was statistically significantly lower than the two 
highest participation groups. The two highest participation groups (60-89 days, and 90+ days) 
had statistically significantly higher attendance rates compared to the two lowest participation 
groups (1-29 days and 30-59 days) overall, and for the subgroups female, ED, and minority. 
Finally, the highest participation group (90+ days) had statistically significantly higher 
attendance than the next highest participation group (60-89 days) for the SPED and ED 
subgroups. 

Between years, 218'CCLC participants in the lowest participation group (1-29 days) were 
the only group to have statistically significant decreases in school-day attendance rates, both 


overall as well as foreach subgroup. The 30-59 days participation group for LEP, and the 60-89 
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days participation group for ED and Minority were the only groups to have statistically 
significant increases between years. 

For the 2018-2019 school year, overall, students in the highest 218'CCLC participation 
group (90+ days) on average, had an attendance rate that was slightly less than 1.5 percentage 
points higher compared to students in the lowest 218' CCLC attendance group (1-29 days). 
Furthermore, this difference increased slightly to almost 2 percentage points in 2019-2020. The 
largest statistically significant differences between 215'CCLC participation groups were between 
the lowest and highest participation groups in the 2019-2020 school year for the SPED and LEP 
subgroups. For these two subgroups, students who participated in 218'CCLC programs for 90+ 
days had on average, an attendance rate that was 2.5 percentage points higher compared to 
students who participated between 1-29 days. 

Readers should note that the 1-29 days participation group was much larger than the other 
participation groups, making up approximately two-thirds to over 80% of the overall samples in 
both years. Conversely, the 90+ days participation group made up approximately only 2%-3% of 
the overall sample. Therefore, comparisons between participation groups should be treated with 
caution due to the large differences in sample sizes. In addition, due to the fact that schools 
closed in March 2020 due to the COVID-19 pandemic, it is not possible to know if attendance 


data for the year would have been different had it been a normal, full school year. 


What promising practices regarding the achievement of required objectives were identified by 
centers? 

Among comments submitted by grantees across the six subjects (math and 
reading/language arts; parent education; character education; enrichment opportunities; and 
community partnerships), the most heavily emphasized “promising practices” addressed three 
broad areas. First and most prominently were practices that supported the students. These can be 
broken into three types: Support for academic performance; enrichment activities; and use of 
small groups. The second broad group of practices encompassed family engagement through 
outreach, communication, and relationship building. Finally, there were practices such as 
frequent, consistent, and/or continuous communication aimed at improving community 
partnerships. 

Grantees were also asked for recommendations to improve the program in the future. 


Most recommendations were for practices already mentioned in responses about the six subjects 
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listed above. Among ideas not already emphasized were recommended practices under the broad 
themes of family communication and engagement, program structure due to COVID 19, and 


staff retention. 
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